---
title: "App WebSocket"
description: "WebSocket endpoint for third-party app connections"
---

## Endpoint

<CodeGroup>
```bash Production
wss://api.mentra.glass/app-ws
```

```bash Development
wss://devapi.mentra.glass/app-ws
```

```bash Local
ws://localhost:8002/app-ws
```
</CodeGroup>

## Authentication

Apps authenticate using a JWT token in the Authorization header, along with session information in custom headers.

### Headers

```typescript
{
  "Authorization": "Bearer <appJwt>",
  "x-user-id": "user@example.com",
  "x-session-id": "session-123",
  "Host": "cloud.mentraos.com",
  "Upgrade": "websocket",
  "Connection": "Upgrade"
}
```

### JWT Token Structure

The appJwt must contain:
- `packageName`: The app's package identifier
- `apiKey`: The app's API key

## Connection Flow

<Steps>
  <Step title="Webhook Notification">
    App receives webhook when user starts the app
  </Step>
  <Step title="WebSocket Connection">
    App connects to WebSocket with JWT and session headers
  </Step>
  <Step title="Authentication">
    Server validates JWT token and verifies API key
  </Step>
  <Step title="Session Binding">
    App is bound to the user's active session
  </Step>
  <Step title="Connection Init">
    App sends CONNECTION_INIT to complete handshake
  </Step>
</Steps>

## Message Types

### App → Cloud Messages

All message types are defined in `packages/sdk/src/types/messages/app-to-cloud.ts`:

#### CONNECTION_INIT
```json
{
  "type": "CONNECTION_INIT",
  "packageName": "com.example.app",
  "sessionId": "session-123",
  "apiKey": "app_api_key_here"
}
```

#### SUBSCRIPTION_UPDATE
```json
{
  "type": "SUBSCRIPTION_UPDATE",
  "subscriptions": [
    {
      "type": "transcription",
      "config": {
        "languages": ["en-US", "es-ES"],
        "interimResults": true
      }
    },
    {
      "type": "button_press",
      "config": {
        "buttons": ["main"]
      }
    }
  ]
}
```

#### DISPLAY_REQUEST
```json
{
  "type": "DISPLAY_REQUEST",
  "packageName": "com.example.app",
  "view": "MAIN",
  "layout": {
    "type": "TEXT_WALL",
    "text": "Hello from app!"
  },
  "durationMs": 5000,
  "forceDisplay": false
}
```

#### PHOTO_REQUEST
```json
{
  "type": "PHOTO_REQUEST",
  "requestId": "photo-456",
  "saveToGallery": false,
  "customWebhookUrl": "https://app.example.com/photo-webhook"
}
```

#### AUDIO_PLAY_REQUEST
```json
{
  "type": "AUDIO_PLAY_REQUEST",
  "requestId": "audio-789",
  "audioUrl": "https://example.com/sound.mp3",
  "volume": 0.8
}
```

### Cloud → App Messages

All message types are defined in `packages/sdk/src/types/messages/cloud-to-app.ts`:

#### CONNECTION_ACK
```json
{
  "type": "CONNECTION_ACK",
  "settings": [],           // App-specific settings
  "mentraosSettings": {},   // System settings
  "config": {},             // App config
  "capabilities": {
    "hasDisplay": true,
    "hasCamera": true,
    "hasMicrophone": true,
    "hasSpeaker": true
  }
}
```

#### DATA_STREAM
```json
{
  "type": "DATA_STREAM",
  "streamType": "transcription",
  "data": {
    "type": "transcription",
    "text": "Hello world",
    "isFinal": true,
    "transcribeLanguage": "en-US",
    "startTime": 1234567890,
    "endTime": 1234567892,
    "confidence": 0.95
  }
}
```

#### SETTINGS_UPDATE
```json
{
  "type": "SETTINGS_UPDATE",
  "settings": {
    "theme": "dark",
    "language": "en-US"
  }
}
```

#### APP_STOPPED
```json
{
  "type": "APP_STOPPED",
  "reason": "USER_REQUEST"
}
```

## Subscription System

Apps must subscribe to the data streams they want to receive. Available stream types are defined in `packages/sdk/src/types/streams.ts`:

### Hardware Streams
- `button_press` - Button press events
- `head_position` - Head position changes
- `location_update` - GPS location updates
- `vps_coordinates` - Visual positioning system

### Audio Streams
- `transcription` - Speech-to-text results
- `translation` - Real-time translation
- `VAD` - Voice activity detection
- `audio_chunk` - Raw audio data

### Phone Streams
- `phone_notification` - Incoming notifications
- `phone_notification_dismissed` - Dismissal events
- `calendar_event` - Calendar updates

### System Streams
- `start_app` - App start requests
- `stop_app` - App stop requests
- `core_status_update` - System status

### Media Streams
- `photo_taken` - Photo capture events
- `rtmp_stream_status` - RTMP streaming status
- `managed_stream_status` - Managed streaming

## Error Handling

### Authentication Errors

```json
{
  "type": "CONNECTION_ERROR",
  "error": "Invalid API key"
}
```

### Subscription Errors

```json
{
  "type": "SUBSCRIPTION_ERROR",
  "error": "Invalid stream type: unknown_stream"
}
```

## Heartbeat

Apps must send heartbeat messages to maintain their connection:

```json
{
  "type": "HEARTBEAT",
  "timestamp": 1234567890
}
```

If no heartbeat is received for 20 seconds, the app enters a grace period before being disconnected.

## Implementation Details

- **Service**: `AppWebSocketService` (packages/cloud/src/services/websocket/websocket-app.service.ts)
- **Manager**: `AppManager` handles app lifecycle and state
- **Authentication**: API key validation against app database
- **Subscriptions**: `SubscriptionService` manages data stream routing

## SDK Usage

The MentraOS SDK handles all WebSocket communication automatically:

```typescript
import { AppSession } from '@mentraos/sdk';

const session = new AppSession({
  packageName: 'com.example.app',
  apiKey: process.env.MENTRAOS_API_KEY
});

// Connect to session
await session.connect(sessionId, userId);

// Subscribe to events
session.events.onTranscription((data) => {
  console.log('User said:', data.text);
});

// Send display request
await session.layouts.showTextWall('Hello World!');
```